Variogram-based proper scoring rules for probabilistic forecasts

Proper scoring rules provide a theoretically principled framework for the quantitative assessment of the predictive performance of probabilistic forecasts. While a wide selection of such scoring rules for univariate quantities exists, there are only few scoring rules for multivariate quantities, and many of them require that forecasts are given in the form of a probability density function. The energy score, a multivariate generalization of the continuous ranked probability score, is the only commonly used score that is applicable in the important case of ensemble forecasts, where the multivariate predictive distribution is represented by a finite sample. Unfortunately, its ability to detect incorrectly specified correlations between the components of the multivariate quantity is somewhat limited. In this paper we present an alternative class of proper scoring rules based on the geostatistical concept of variograms. We study their sensitivity to incorrectly predicted means, variances, and correlations in a number of examples with simulated observations and forecasts, and show that the variogram-based scoring rules are distinctly more discriminative with respect to the correlation structure. This conclusion is confirmed in a case study with post-processed wind speed forecasts at five wind park locations in Colorado, U.S.A. 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26